Methods and Systems to Modify a Two Dimensional Facial Image to Increase Dimensional Depth and Generate a Facial Image That Appears Three Dimensional

ABSTRACT

The specification describes methods and systems for increasing a dimensional depth of a two-dimensional image of a face to yield a face image that appears three dimensional. The methods and systems identify key points on the 2-D image, obtain a texture map for the 2-D image, determines one or more proportions within the 2-D image, and adjusts the texture map of the 3-D model based on the determined one or more proportions within the 2-D image.

FIELD

The present specification discloses systems and methods for modifying afacial image. More specifically, the present specification is related tomodifying a two-dimensional (2-D) facial image to obtain a correspondingmodified facial image that has an increased dimensional depth, therebyappearing to be a three-dimensional (3-D) facial image. The 3D facialimage can then be integrated into a game in the form of an avatar anddisplayed in a graphical user interface.

BACKGROUND

A three-dimensional (3-D) image provides a perception of depth, and canbe used in a variety of virtual environments. A virtual environmentprovides an interactive experience, in the form of virtual reality, to auser. 3-D imagery is becoming intensely popular in virtual environmentsthat are experienced on screens, or special display devices such as headmounted devices or goggles. 3-D imagery is also used in gaming systems,simulations, architectural walkthroughs, and in several other scenarios.

The process of creating and displaying three-dimensional (3D) objects inan interactive computer environment is a complicated matter. Thecomplexity increases with the need to convert a 2-D image to acorresponding 3-D model. A 2-D image includes two axes, whereas a 3-Dimage incorporates a third axis, which provides the depth component. Itshould be appreciated that the 3-D image is still being displayed on atwo dimensional display but it has been modified, relative to the 2-Dimage, to include a dimensional depth that, when viewed by a user, makesthe flat, planar image visually appear to be three dimensional.

Commonly available methods that convert a 2-D image to a corresponding3-D model require combining multiple images that provide multiple viewsof the 2-D image. For example, a front view photo and a side view photoof a face may be required to recreate the face in 3-D. Some methodsrequire specialized software programs to covert one or multiple 2-Dinput images to a 3-D output model. Yet other methods require atechnician to work with specialized software programs to convert a 2-Dimage in to a corresponding 3-D model. These methods may significantlyincrease computational complexity, or may require time-consuming manualinterventions in adjusting and/or aligning 2-D image(s) to create acorresponding 3-D model. Moreover, computerized methods of converting2-D images of faces, such as faces of humans, pose several limitationsin understanding the human face and features that vary widely with eachindividual. Some other methods, such as UV mapping, involve projecting a2-D image on to a 3-D model surface to obtain texturized 2-D image.However, these methods are unable to match specific facial features fromthe 2-D image of a face to the corresponding 3-D mesh model.

There is a need for improved, automated methods and systems forconverting a single 2-D image to a corresponding image with increaseddimensional depth to create an image that appears 3-D. There is also aneed for improved, automated methods and systems for converting a single2-D image to a corresponding 3-D image in substantially real time, whichcan overcome the above limitations and disadvantages of the currentmethods.

SUMMARY

The following embodiments and aspects thereof are described andillustrated in conjunction with systems, tools and methods, which aremeant to be exemplary and illustrative, not limiting in scope.

In some embodiments, the present specification discloses acomputer-implemented method for increasing a dimensional depth of atwo-dimensional image of a face to yield a face image that appears threedimensional, said method being implemented in a computer having aprocessor and a random access memory, wherein said processor is in datacommunication with a display and with a storage unit, the methodcomprising: acquiring from the storage unit the two-dimensional image ofthe face; acquiring from the storage unit a three-dimensional meshimage; using said computer and executing a plurality of programmaticinstructions stored in the storage unit, identifying a plurality of keypoints on the two-dimensional image of the face; using said computer andexecuting a plurality of programmatic instructions stored in the storageunit, generating a texture map of the two-dimensional image of the face;using said computer and executing a plurality of programmaticinstructions stored in the storage unit, projecting said texture map ofthe two-dimensional image of the face onto the three-dimensional meshimage; using said computer and executing a plurality of programmaticinstructions stored in the storage unit, determining a first set of oneor more proportions within the two-dimensional image of the face; usingsaid computer and executing a plurality of programmatic instructionsstored in the storage unit, determining a second set of one or moreproportions within the three-dimensional mesh image; using said computerand executing a plurality of programmatic instructions stored in thestorage unit, determining a plurality of scaling factors, wherein eachof said scaling factors is a function of one of said first set of one ormore proportions and a corresponding one of said second set of one ormore proportions; using said computer and executing a plurality ofprogrammatic instructions stored in the storage unit, adjusting thethree-dimensional mesh image based on the determined plurality ofscaling factors to yield the face image that appears three dimensional;and using said computer, outputting the face image that appears threedimensional.

In some embodiments, the key points may include points representative ofa plurality of anatomical locations on the face, wherein said anatomicallocations include points located on the eyebrows, eyes, nose, and lips.

Optionally, the texture map comprises a plurality of non-overlapping,triangular regions.

Optionally, each of said plurality of scaling factors is a ratio of oneof said first set of one or more proportions to the corresponding one ofsaid second set of one or more proportions.

In some embodiments, the determining the first set of one or moreproportions within the two-dimensional image may comprise determiningproportions from measurements between at least two anatomical positionson the face.

In some embodiments, the determining a first set of one or moreproportions within the two-dimensional image may comprise determining afirst anatomical distance and dividing said first anatomical distance bya second anatomical distance.

Optionally, the first anatomical distance is at least one of a lateralface width, a lateral jaw width, a lateral temple width, a lateraleyebrow width, a lateral chin width, a lateral lip width, and a lateralnose width and wherein the second anatomical distance is a distancebetween two temples of the face. Still optionally, the first anatomicaldistance is at least one of a vertically defined lip thickness, avertical distance between a nose and a nose bridge, a vertical distancebetween a lip and a nose bridge, a vertical distance between a chin anda nose bridge, a vertical eye length, and a vertical distance between ajaw and a nose bridge and wherein the second anatomical distance is atleast one of a distance between two anatomical positions on said faceand a distance between two temples of the face. Still optionally, thefirst anatomical distance is a distance between two anatomical positionson said face and the second anatomical distance is a distance between apoint located proximate a left edge of a left eyebrow of the face and apoint located proximate a right edge of a right eyebrow of the face.

Optionally, the determining a second set of one or more proportionswithin the three-dimensional mesh image comprises determining a firstanatomical distance and dividing said first anatomical distance by asecond anatomical distance.

Optionally, the first anatomical distance is at least one of a lipthickness, a distance between a nose and a nose bridge, a distancebetween a lip and a nose bridge, a distance between a chin and a nosebridge, an eye length, and a distance between a jaw and a nose bridge ofthe three-dimensional mesh image and wherein the second anatomicaldistance is a distance between two anatomical positions on saidthree-dimensional mesh image. Still optionally, the first anatomicaldistance is a distance between two anatomical positions on saidthree-dimensional mesh image and the second anatomical distance is adistance between a point located proximate a left edge of a left eyebrowof the three-dimensional mesh image and a point located proximate aright edge of a right eyebrow of the three-dimensional mesh image.

In some embodiments, the computer-implemented method may process thetwo-dimensional image to validate a presence of a frontal image of theface prior to identifying the plurality of key points on thetwo-dimensional image of the face.

In some embodiments, the present specification discloses a computerreadable non-transitory medium comprising a plurality of executableprogrammatic instructions wherein, when said plurality of executableprogrammatic instructions are executed by a processor, a process forincreasing a dimensional depth of a two-dimensional image of a face toyield a face image that appears three dimensional is performed, saidplurality of executable programmatic instructions comprising:programmatic instructions, stored in said computer readablenon-transitory medium, for acquiring from the storage unit thetwo-dimensional image of the face; programmatic instructions, stored insaid computer readable non-transitory medium, for acquiring from thestorage unit a three-dimensional mesh image; programmatic instructions,stored in said computer readable non-transitory medium, for identifyinga plurality of key points on the two-dimensional image of the face;programmatic instructions, stored in said computer readablenon-transitory medium, for generating a texture map of thetwo-dimensional image of the face; programmatic instructions, stored insaid computer readable non-transitory medium, for translating saidtexture map of the two-dimensional image of the face onto thethree-dimensional mesh image; programmatic instructions, stored in saidcomputer readable non-transitory medium, for determining a first set ofone or more proportions within the two-dimensional image of the face;programmatic instructions, stored in said computer readablenon-transitory medium, for determining a second set of one or moreproportions within the three-dimensional mesh image; programmaticinstructions, stored in said computer readable non-transitory medium,for determining a plurality of scaling factors, wherein each of saidscaling factors is a function of one of said first set of one or moreproportions and a corresponding one of said second set of one or moreproportions; and programmatic instructions, stored in said computerreadable non-transitory medium, for adjusting the three-dimensional meshimage based on the determined plurality of scaling factors to yield theface image that appears three dimensional.

Optionally, the key points include points representative of a pluralityof anatomical locations on the face, wherein said anatomical locationsinclude points located on the eyebrows, eyes, nose, and lips.

Optionally, the texture map comprises a plurality of non-overlapping,triangular regions.

Optionally, the determining one or more proportions within thetwo-dimensional image comprises determining proportions frommeasurements between at least two anatomical positions on the face.

Optionally, each of said plurality of scaling factors is a ratio of oneof said first set of one or more proportions to the corresponding one ofsaid second set of one or more proportions.

Optionally, the determining a first set of one or more proportionswithin the two-dimensional image comprises determining a firstanatomical distance and dividing said first anatomical distance by asecond anatomical distance.

Optionally, the first anatomical distance is at least one of a lateralface width, a lateral jaw width, a lateral temple width, a lateraleyebrow width, a lateral chin width, a lateral lip width, and a lateralnose width and wherein the second anatomical distance is a distancebetween two temples of the face.

Optionally, the first anatomical distance is at least one of avertically defined lip thickness, a vertical distance between a nose anda nose bridge, a vertical distance between a lip and a nose bridge, avertical distance between a chin and a nose bridge, a vertical eyelength, and a vertical distance between a jaw and a nose bridge andwherein the second anatomical distance is a distance between two templesof the face.

Optionally, the first anatomical distance is a distance between twoanatomical positions on said face and the second anatomical distance isa distance between a point located proximate a left edge of a lefteyebrow of the face and a point located proximate a right edge of aright eyebrow of the face.

Optionally, the determining a second set of one or more proportionswithin the three-dimensional mesh image comprises determining a firstanatomical distance and dividing said first anatomical distance by asecond anatomical distance.

Optionally, the first anatomical distance is at least one of a lipthickness, a distance between a nose and a nose bridge, a distancebetween a lip and a nose bridge, a distance between a chin and a nosebridge, an eye length and a distance between a jaw and a nose bridge ofthe three-dimensional mesh image and wherein the second anatomicaldistance is a distance between two anatomical positions on saidthree-dimensional mesh image.

Optionally, the first anatomical distance is a distance between twoanatomical positions on said three-dimensional mesh image and the secondanatomical distance is a distance between a point located proximate aleft edge of a left eyebrow of the three-dimensional mesh image and apoint located proximate a right edge of a right eyebrow of thethree-dimensional mesh image.

Optionally, the computer readable non-transitory medium furthercomprises programmatic instructions, stored in said computer readablenon-transitory medium, for processing the two-dimensional image tovalidate a presence of a frontal image of the face prior to identifyingthe plurality of key points on the two-dimensional image of the face.

The aforementioned and other embodiments of the present invention shallbe described in greater depth in the drawings and detailed descriptionprovided below.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features and advantages of the present invention will beappreciated, as they become better understood by reference to thefollowing detailed description when considered in connection with theaccompanying drawings, wherein:

FIG. 1 illustrates the sequence of processing an image to identify a setof key points, in accordance with some embodiments of the presentspecification;

FIG. 2 illustrates a first step, where an image with multiple key pointsis used to define non-overlapping, three-point regions, in accordancewith some embodiments of the present specification;

FIG. 3 illustrates a step where a texture map is obtained for a 3-Dmodel, in accordance with some embodiments of the present specification;

FIG. 4 illustrates processing of the images to arrive at an image, whichis a texture map of the 3-D model created using triangulation of the keypoints, in accordance with some embodiments of the presentspecification;

FIG. 5 illustrates various horizontal (green lines) and vertical (redlines) measurements with the original image that may be used inaccordance with some embodiments of the present specification;

FIG. 5A illustrates a placement of various key points on an exemplaryface, in accordance with some embodiments of the present specification;

FIG. 5B illustrates the various key points of FIG. 5A without theexemplary face, in accordance with some embodiments of the presentspecification;

FIG. 6 illustrates a generic 3-D mesh model, to which scale factors areapplied, in order to obtain a modified 3-D mesh model, in accordancewith some embodiments of the present specification;

FIG. 7 is a flow chart illustrating an exemplary computer-implementedmethod to convert a 2-D image in to a 3-D image for display, inaccordance with embodiments of the present specification;

FIG. 8 is a flow chart illustrating an exemplary computer-implementedmethod to convert a 2-D image in to a 3-D image for display as anavatar, in accordance with embodiments of the present specification;

FIG. 9 illustrates a set of images that describe the conversion of a 2-Dfrontal face image to a 3-D avatar, in accordance with some embodimentsof the present specification;

FIG. 10 is a flow chart illustrating an exemplary computer-implementedmethod to convert a 2-D image in to a 3-D image for display using ARmasks, in accordance with embodiments of the present specification;

FIG. 11 illustrates multiple AR images, created with the method outlinedwith reference to FIG. 10, in accordance with some embodiments of thepresent specification;

FIG. 12 is a flow chart illustrating an exemplary computer-implementedmethod to convert a 2-D image in to a 3-D image for display of facelessgaming and other interactive display characters, which are controlled byreal individuals, in accordance with some embodiments of the presentspecification;

FIG. 13 illustrates exemplary images captured from gaming displays, inaccordance with some embodiments of the present specification;

FIG. 14 is a flow chart illustrating an exemplary computer-implementedmethod to convert a 2-D image in to a 3-D image for display of playersin gaming environments and other interactive display characters, whichare controlled by real individuals, in accordance with some embodimentsof the present specification;

FIG. 15 illustrates a still image from a display of a gamingapplication, where a 3-D image of a user (player) is seen in thedisplay, in accordance with some embodiments of the presentspecification;

FIG. 16 is a flow chart illustrating an exemplary computer-implementedmethod to convert a 2-D image of an individual in to a 3-D image thatmay be used to print one or more personalized avatars of the individual,in accordance with some embodiments of the present specification;

FIG. 17 illustrates some examples of different (four) avatars ofdifferent users printed using embodiments of the process described incontext of FIG. 16; and

FIG. 18 is a flow chart illustrating an exemplary computer-implementedmethod to convert a 2-D image in to a 3-D image for driving key frameanimation based on facial expressions of an individual, in accordancewith some embodiments of the present specification.

DETAILED DESCRIPTION

In an embodiment, a method is provided for converting a two-dimensional(2-D) image for three-dimensional (3-D) display using a computingdevice, such as a laptop, mobile phone, desktop, tablet computer, orgaming console, comprising a processor in data communication with anon-transient memory that stores a plurality of programmaticinstructions which, when executed by the processor, perform the methodsof the present invention. The 2-D image may be in any known format,including, but not limited to, ANI, ANIM, APNG, ART, BMP, BPG, BSAVE,CAL, CIN, CPC, CPT, DDS, DPX, ECW, EXR, FITS, FLIC, FLIF, FPX, GIF,HDRi, HEVC, ICER, ICNS, ICO/CUR, ICS, ILBM, JBIG, JBIG2, JNG, JPEG, JPEG2000, JPEG-LS, JPEG XR, KRA, MNG, MIFF, NRRD, ORA, PAM, PBM/PGM/PPM/PNM,PCX, PGF, PlCtor, PNG, PSD/PSB, PSP, QTVR, RAS, RBE, SGI, TGA, TIFF,UFO/UFP, WBMP, WebP, XBM, XCF, XPM, XWD, CIFF, DNG, AI, CDR, CGM, DXF,EVA, EMF, Gerber, HVIF, IGES, PGML, SVG, VML, WMF, Xar, CDF, DjVu, EPS,PDF, PICT, PS, SWF, XAML and any other raster, raw, vector, compound, orother file format.

In embodiments, the conversion from a 2-D image to a modified image withincreased dimensional depth to thereby appear to be 3-D, generallyreferred to as a 3-D image, is performed automatically after the 2-Dimage is obtained by the computing device. In an embodiment, a single2-D image is processed to identify key points of interest. These pointsare used to define three-point regions that are exclusive of each other.In an embodiment, a Delaunay triangulation method is used to define thethree-point regions automatically. The triangulation is used tosynchronize with pre-indexed points of interest laid out on a 3-D model,thereby enabling a UV mapping of the 2-D image to yield a texturized 3-Dmodel. In various embodiments, proportions and ratios that are unique tothe 2-D image and the texturized 3-D model are used to calculate atleast one scale factor. The scale factors are used to sculpt the 3-Dimage corresponding to the original 2-D image.

The present specification is directed towards multiple embodiments. Thefollowing disclosure is provided in order to enable a person havingordinary skill in the art to practice the invention. Language used inthis specification should not be interpreted as a general disavowal ofany one specific embodiment or used to limit the claims beyond themeaning of the terms used therein. The general principles defined hereinmay be applied to other embodiments and applications without departingfrom the spirit and scope of the invention. Also, the terminology andphraseology used is for the purpose of describing exemplary embodimentsand should not be considered limiting. Thus, the present invention is tobe accorded the widest scope encompassing numerous alternatives,modifications and equivalents consistent with the principles andfeatures disclosed. For purpose of clarity, details relating totechnical material that is known in the technical fields related to theinvention have not been described in detail so as not to unnecessarilyobscure the present invention. In the description and claims of theapplication, each of the words “comprise” “include” and “have”, andforms thereof, are not necessarily limited to members in a list withwhich the words may be associated.

It should be noted herein that any feature or component described inassociation with a specific embodiment may be used and implemented withany other embodiment unless clearly indicated otherwise.

FIG. 1 illustrates the sequence of processing an image with a generalpurpose cross-platform software library that contains machine learningalgorithms. Exemplary features in the software library enables thedetection of face features, including eyebrows, eyes, nose, mouth,nostrils, ears, cheekbones, chin, and/or lips. It should be appreciatedthat any facial feature detection software may be implemented, providedthe software detects more than one location on (and assigns a distinctpoint to more than one location) on each of the two eyebrows, two eyes,nose, and lips of the face and assigns each of those distinct pointswith a distinct horizontal position (e.g., X position) and verticalposition (e.g., Y position) in a coordinate system.

Referring back to FIG. 1, a facial image 102 is processed using asoftware application. In various embodiments, the image 102 is sourcedfrom a viewing element such as a camera, a database of images, a video,a memory local to the computing device, a memory remote from thecomputing device, or any other source of images. In one embodiment, theimage 102 is a result of a selfie shot taken by an individual through acamera of a mobile phone and stored locally within the mobile phone.Processing validates that the image is a frontal image of a face. Afrontal image of a face may be best suited in a display for gamingapplications and several other virtual reality, augmented reality ormixed reality applications. The remaining components (body parts) of adisplay created for an individual may be created and presented invarious imaginative formats.

Using facial feature detection software, the frontal face portion 104may thus be isolated from the remaining image. Optionally, the facialimage may be analysed to determine if the image is sufficiently“front-facing”. More specifically, if the facial image is too skewed,whether up, down, diagonally, left, right, or otherwise, the presentlydisclosed methods and systems may have a difficult time generating aquality three dimensional image. As a result, in one embodiment, thepresently disclosed system analyses the image to determine if the faceis turned greater than a predefined angle, if the edges of the face aresubstantially the same distance from the center of the face, and/or ifthe features on one side of the face, such as lips, eyes or ears, aredimensionally different, in excess of a predefined threshold, relativeto the features on the other side of the face.

If the facial image is sufficiently “front-facing”, subsequently, thesystem identifies multiple key anatomical points, as seen in image 106,which indicate anatomically defined features of the face in image 102. Akey anatomical point is a location on a face that is detected andprovided by a software application. An exemplary software applicationuses a face detection function that returns a list of 67 points on theface (in pixels). In embodiments of the present specification, thesystem numbers key points in image 106, as seen in image 108. Image 108illustrates key points indexed up to 67 and the numbers shown in image108 indicate an assigned identity (ID) of each key point. It should beappreciated that the system may identify any number of anatomical pointsthat may be less than or greater than 67.

Subsequently, the system generates a texture map for image 106. Thesystem generates a texture map using two steps. FIG. 2 illustrates afirst step, where an image 206 with multiple key points is used todefine a plurality of non-overlapping regions, each of which may bedefined by at least three points, as seen in image 210. In anembodiment, the regions define various anatomical regions, therebycapturing various anatomical features, of a front-face of a human being.In an embodiment, at least the anatomical regions that define brows,eyes, nose, lips, and face, are covered by the key points. Inembodiments, the non-overlapping, three point, triangular regions aredefined using Delaunay triangulation, automatically through theexecution of programmatic instructions on the computing device. Use ofDelaunay triangulation, which is a known analytical method to persons ofordinary skill in the art, ensures that none of the key points areinside the circumcircle of any triangular region, thus maximizing theminimum angle of all the angles of the triangles in the triangulation.Maximizing the minimum angles thereby improves subsequent interpolationor rasterization processes that may be applied to the image for creatinga corresponding 3-D mesh.

Referring to FIG. 3, the system then initiates a process of generating atexture map for a 3-D model. Triangulation from the previous step isautomatically used to synchronize a plurality of pre-defined and/orpre-indexed points of interest identified on a UV layout. In analternative embodiment, the points of interest are automaticallygenerated by the system. In embodiments, the points of interestidentified for the UV layout are termed as landmark points of UV map. AUV layout is a layout of a 2D model where U and V refer to the two axes(corresponding to X and Y axes in the 3D model). In embodiments, thesystem stores, in a memory, a generic 2D UV layout and its correspondinggeneric 3D model. The system uses the triangulation data to modify thegeneric UV layout and uses the modified UV layout to create a modified3D model.

In FIG. 3, an image 302 illustrates a pre-defined or automaticallygenerated generic 3-D mesh model. An image 304 illustrates the texturecoordinates or the landmark points on the UV map of the generic 3-D meshmodel. The texture coordinates or landmark points are generated for eachvertex of the triangles derived through the triangulation. Inembodiments, image 304 is generated automatically by a softwareapplication. Image 306 illustrates the triangles derived through thetriangulation process for the UV map of the generic 3D model.

FIG. 4 illustrates processing of the images to arrive at an image 406,which is a texture map of the 3-D model. The figure shows an image 402corresponding to the original image 104, with triangulations of the keypoints. Images 402 of FIG. 4 and 210 of FIG. 2 illustrate a resultingcollection of tri-faces within the original image 104. The systemmatches the tri-faces of image 402 with corresponding triangles withinthe UV map 404 of the generic 3-D mesh model. In an embodiment, this isachieved by matching each key point of the original image with thetexture coordinates or landmark points of the UV map. Thus, the systemuses the key points to morph the generic UV layout such that the UVlayout is modified in to a front-face image 406, which includes thetexture of the original image of the face. The texture includes colorsand other visual parameters, such as hue, luminance, brilliance,contrast, brightness, exposure, highlights, shadows, black point,saturation, intensity, tone, grain, neutrals; which are automaticallysampled from various parts of the face in the original image andaveraged, or subjected to some other processing formula, to fill out thebackdrop within the morphed 3-D mesh image. The resultant image may beused as a texture map for the 3D model.

Various embodiments of the present specification enable accounting forthe unique proportions and ratios of the original image. In the givenexample, the original image 104 used for processing a frontal face of anindividual, is additionally used to identify face feature lengths,distances, proportions, dimensions, or ratios, collectively referred toas positional relationships. In embodiments, the system analyses image104 to generate values indicative of the positional relationships of anindividual's facial features. For example, the values may berepresentative of the relative distances between width of the nosecompared to the chin, distance between the two eyes, width of eyebrows,thickness of the lips, and other measurements that mark the positionalrelationships of various anatomical points and/or regions on the face.

In an embodiment, the system determines a plurality of distances betweenvarious anatomical facial features. In embodiments, the distances areused to adjust the generated 3D model of the original image of the face.Referring to FIG. 5, image 502 illustrates various horizontal (greenlines) and vertical (red lines) measurements with the original image(104) that may be used in accordance with some embodiments of thepresent specification, to identify the positional relationships. Image504 illustrates various measurements of similar features of the generic3-D mesh model face (image 302 of FIG. 3).

FIGS. 5A and 5B show an enlarged view of key points 1 to 67 521,identified through a plurality of programmatic instructions configuredto graphically identify a plurality of key points. The number IDs forthe key points 521 shown in FIGS. 5A and 5B are recalled (in brackets)in the following examples. The system uses the corresponding similarmeasurements to derive positional relationships between the measuredfeatures. One of the plurality of exemplary distances is a face width,defined as a distance 508 (2 and 14) from one point located proximatethe left edge of the face laterally across the face to a point locatedproximate on the right edge of the face. Another of the plurality ofexemplary distances is a jaw width, defined as a distance 510 (3 and 13)from one point located proximate the left edge of the jaw laterallyacross the face to a point located proximate on the right edge of thejaw. Yet another of the plurality of exemplary distances is a templewidth, defined as a distance 512 (0 and 16) from one point locatedproximate the left edge of the temple laterally across the face andthrough the eyes, to a point located proximate on the right edge of thetemple. Still another of the plurality of exemplary distances is aneyebrow width, defined as a distance 514 (22 and 26) from one pointlocated proximate the left edge of an eyebrow, laterally across thewidth of the eyebrow, to a point located in line with the right edge ofthe same eyebrow. Another of the plurality of exemplary distances is achin width, defined as a distance 516 (7 and 9) from one point locatedproximate the left edge of the chin laterally across the chin, to apoint located proximate on the right edge of the chin. Another of theplurality of exemplary distances is a lip width, defined as a distance518 (48 and 64) from one point located proximate the left corner of thelips where the upper and the lower lip meet, across the mouth, to apoint located proximate on the right corner of the lips. Another of theplurality of exemplary distances is a nose width, defined as a distance520 (31 and 35) from one point located proximate the left edge of theleft opening of the nose, to a point located proximate on the right edgeof the right opening of the nose. In embodiments, additional and/orother combinations of horizontal distances between various anatomicalpoints mapped laterally across the face are used to obtain thepositional relationships.

Distances may also be measured vertically across the length of the face.One of the plurality of exemplary distances is a lip thickness, definedas a distance 522 (51 and 57) from one point located proximate thecentre of a top edge of an upper lip, vertically across the mouth, to apoint located proximate the centre of a bottom edge of a lower lip.Another one of the plurality of exemplary distances is a distancebetween nose and nose bridge, defined as a distance 524 (27 and 33) fromone point located proximate the centre of the eyes where the top of anose bridge is positioned, vertically across the nose, to a pointlocated proximate the centre of the nose openings. Yet another of theplurality of exemplary distances is a distance between lip and nosebridge, defined as a distance 526 (27 and 66) from one point locatedproximate the centre of the eyes where the top of a nose bridge ispositioned, vertically across the nose and the upper lip, to a pointlocated proximate the center of the mouth. Still another of theplurality of exemplary distances is a distance between chin and nosebridge, defined as a distance 528 (27 and 8) from one point locatedproximate the centre of the eyes where the top of a nose bridge ispositioned, vertically across the nose, the upper lip, and the mouth, toa point located proximate the centre of the chin. Another of theplurality of exemplary distances is an eye length, defined as a distance530 (44 and 46) from one point located proximate the centre of a top ofan eye, vertically across the eye, to a point located proximate thecentre of a bottom of the eye. Another of the plurality of exemplarydistances is an eyebrow height, defined as a distance 532 (24 and 44)from one point located proximate the centre of the eyebrow, verticallyacross the eye, to a point located proximate the centre of the eye underthe eyebrow. Another of the plurality of exemplary distances is a jawand nose bridge distance, defined as a distance 534 (27 and 3) from onepoint located proximate the centre of the nose bridge, vertically acrossthe length of the cheek, to a point located proximate the jaw. Inembodiments, additional and/or other combinations of vertical distancesbetween various anatomical points mapped vertically across the face areused to obtain the positional relationships.

In embodiments, additional and/or other combinations of diagonaldistances between various anatomical points mapped laterally across theface are used to obtain the positional relationships. An example is adistance 536 (22 and 26) between a point located proximate the left edgeof one eyebrow to the right edge of the same eyebrow, which indicatesthe brow angle.

In embodiments, the system obtains positional relationships in image 502by determining one or more proportions, based on the one or more oflateral, vertical, and diagonal distances. In an exemplary embodiment,face width 508, measured between key points with IDs 2 and 14, is usedas a constant to determine proportions of other measured distances. Forexample, one of the plurality of proportions is derived by usingdistance 510 (3 and 13) as the numerator and face width 508 as thedenominator. The exemplary proportion described here provides thepositional relationship of the jaw with respect to the face. In analternative embodiment, the system uses distance 512 between key pointswith ID 0 and with ID 16, which may indicate the entire temple width ofthe facial image, as a whole unit in the denominator to subsequentlycalculate ratios on all the rest of the face. While other anatomicaldistances may be used as the denominator to calculate one or moreproportions, temple width is the preferred distance because it tends toremain predictably static, even if people gain weight, lose weight, age,or undergo collagen or botox injections.

In embodiments, similar proportions are determined for the 3-D meshmodel image 504. As described above in relation to image 502, the systemobtains positional relationships in image 504 by determining one or moreproportions, based on the one or more of lateral, vertical, and diagonaldistances in relation to a standard anatomical distance, such as templewidth.

Once both sets of proportions are obtained, the system uses proportionsfrom both images 502 and 504 to calculate their ratio, in order todetermine scale factors 506. In an embodiment, scale factor 506 is theratio of proportions or positional relationships of image 502, to thecorresponding proportions or positional relationships of image 504.Image 506 illustrates exemplary scale factors derived usingcorresponding proportions from the image of the face of an individual502 and the generic 3-D mesh model 504.

In an embodiment, these measurements are communicated to a console of acomputing and/or a mobile computing device, along with the newlygenerated texture map 406. FIG. 6 illustrates a generic 3-D mesh model602 504, 302 to which the system applies the scale factors 604, 506, inorder to obtain a modified 3-D mesh model 606. In embodiments, thetexture map 406 obtained previously is applied to modified 3-D meshmodel 606, to obtain a final 3-D model 608 of the original 2-D image. Onthe console side, the 3-D model's texture is swapped out with the newlycreated model 606, and the proportion measurements are used to driveadjustments to mirror the individual's actual face structure. This, ineffect, sculpts the mesh to more closely resemble the captured face.

FIG. 7 is a flow chart illustrating an exemplary computer-implementedmethod to convert a 2-D image into a modified image that has increaseddimensional depth (and, therefore, appears three dimensional to a user)for display, in accordance with embodiments of the presentspecification. At 702, the system, according to various embodiments ofthe present specification, identifies key points on the 2-D image. In anembodiment, the system uses a plurality of programmatic instructionsdesigned to graphically identify a plurality of key points, to identifyat least 67 key points. Subsequently the system derives a texture mapfor the 2-D image. The system derives a texture map using the followingsteps. First, at 704, the system identifies a plurality ofnon-overlapping, three-point regions based on the identified key points.In embodiments, a set of pre-defined landmark points of UV map of ageneric 3-D mesh 304 are brought in coherence with the key points on the2-D image 206. Preferably, the landmark points of UV map of the generic3-D mesh are generated automatically. At 705, the system uses Delaunaytriangulation to define the three-point regions, based on the identifiedkey points. The three-point regions, which may also be termed as piecesof tri-faces, represent an average texture of the face within eachregion. Each vertex of the triangles are the UV coordinates (also knownas texture coordinates), which are used to recreate the texture of the2D image on a 3D model. At 706, the system projects the triangulated 2-Dimage on UV map of the generic 3-D mesh model 304.

At 708, the system determines one or more positional relationshipswithin the 2-D image. As described above, the positional relationshipscomprise a plurality of distances between anatomical features in thefacial image, and ratios of those distances to a specific anatomicaldistance such as temple width, which are necessarily unique to the 2-Dimage. Similarly, the system determines one or more positionalrelationships within the generic 3-D mesh model of a face. As describedabove, the positional relationships comprise a plurality of proportionsthat are standard for a generic 3-D face model and comprise a pluralityof distances between anatomical features in the 3-D face model, andratios of those distances to a specific anatomical distance such astemple width, which define the generic 3-D face model.

At 710, the system then uses proportions for the 2-D image and thegeneric 3-D image to determine a ratio, which may be termed as the‘scale factor’. In one embodiment, each scale factor is calculated bytaking a proportion for the 2-D image and dividing it by a proportion ofthe same anatomical features for the 3-D face model. In anotherembodiment, each scale factor is calculated by any function of aproportion for the 2-D image and a proportion of the same anatomicalfeatures for the 3-D face model. It should be appreciated that theaforementioned proportions, for either the 2-D image or 3-D face model,can be determined by taking a distance defining any of the followinganatomical features and dividing it by a distance defining a templewidth: a distance defining lip thickness, a distance between the noseand nose bridge, a distance between a lip and nose bridge, a distancebetween chin and nose bridge, a distance defining an eye length, adistance defining an eyebrow height, and a distance between a j aw andnose bridge distance.

The illustrations of FIG. 1 to FIG. 6 demonstrate the method of FIG. 7implemented on the image of a face. In the exemplary embodiment,proportions may be determined from measurements between at least twoanatomical positions on the face. The anatomical positions may includeanatomical points and/or regions on the face such as but not limited tothe jaw, the nose bridge, the chin, the lip, the eyes, the eyebrow, andother anatomical regions on the face, as described above. At 712, thesystem adjusts proportions of the 3-D mesh model that contains thetexture created till step 706. The proportions are adjusted on the basisof the scale factor determined at step 710, in order to create the 3-Ddisplay for the original 2-D image.

Applications

FIG. 8 is a flow chart illustrating an exemplary computer-implementedmethod to convert a 2-D image into a modified image that has increaseddimensional depth (and, therefore, appears three dimensional to a user)for display as an avatar, in accordance with embodiments of the presentspecification. In embodiments, an avatar of an individual is recreatedin 3-D virtual reality, augmented reality, or mixed reality environment,such as but not limited to gaming environments. In this case, a 2-Dfrontal face image of the individual is used to create a replica of atleast the face in 3-D. The method is similar to the process described inFIG. 7.

At 800, the system obtains an image of the individual from one of thesources including, but not limited to, an independent camera, a cameraintegrated with a mobile or any other computing device, or an imagegallery accessible through a mobile or any other computing device.

At 802, the system, according to various embodiments of the presentspecification, identifies key points on the 2-D image. In an embodiment,the system uses a plurality of programmatic instructions designed tographically identify a plurality of key points, to identify at least 67key points. Subsequently the system derives a texture map for the 2-Dimage. The system derives a texture map using the following steps. At804, the system identifies a plurality of non-overlapping, three-pointregions based on the identified key points. The system uses Delaunaytriangulation to define the three-point regions, based on the identifiedkey points, as described above. At 806, the system projects thetriangulated 2-D image on UV map of the generic 3-D mesh model. At 808,the system determines one or more positional relationships within the2-D image. As described above, the positional relationships comprise aplurality of distances between anatomical features in the facial image,and ratios of those distances to a specific anatomical distance such astemple width, which are necessarily unique to the 2-D image. Similarly,the system determines one or more positional relationships within thegeneric 3-D mesh model of a face. As described above, the positionalrelationships comprise a plurality of proportions that are standard fora generic 3-D face model and comprise a plurality of distances betweenanatomical features in the 3-D face model, and ratios of those distancesto a specific anatomical distance such as temple width, which define thegeneric 3-D face model.

At 810, the system then uses proportions for the 2-D image and thecorresponding proportions from the generic 3-D image to determine thescaling factors. At 812, the system adjusts the 3-D model based on thedetermined scaling factors and at 812, the system creates an avatarusing the 3-D display of the face. The avatar may be used in variousapplications, such as gaming applications. FIG. 9 illustrates a set ofimages that describe the conversion of a 2-D frontal face image 902 to a3-D avatar 906, in accordance with some embodiments of the presentspecification. An image 904 indicates the conversion of image 902 to acorresponding 3-D image (904). In embodiments, image 904 is further usedby computer systems to create an avatar, for example avatar 906.

FIG. 10 is a flow chart illustrating an exemplary computer-implementedmethod to convert a 2-D image into a modified image that has increaseddimensional depth (and, therefore, appears three dimensional to a user)for display using AR masks, in accordance with embodiments of thepresent specification. In embodiments, the masks are animated masks, ormasks created from alternate images. In embodiments, a mask for anindividual is recreated in 3-D virtual reality, augmented reality ormixed reality environments, such as but not limited to online, chattingand gaming environments. In embodiments, the mask is for the face of anindividual and can track the individual's face in real time, therebyfunctioning with the changing facial expressions of the individual. Inembodiments, the mask could be controlled by the individual's facialfeatures. In this case, a 2-D frontal face image of the individual isused to create a replica of at least the face, in 3-D. At 1000, thesystem obtains an image from a database of images. In embodiments, theselected image is an image of a face, an animated character, or a facialexpression created using animation or special effects. In embodiments,the image is obtained from a database of images, a video, a memory localto the computing device, a memory remote from the computing device, orany other source of images. The image obtained at this step issubsequently used as a mask, for example an AR mask, which can beapplied to the face of the individual (user). At 1002, the systemobtains an image of the individual (user) from one of the sourcesincluding, but not limited to, an independent video camera, or a videocamera integrated with a mobile or any other computing device. In anembodiment, the image of the individual is obtained from a webcam. At1004, key points on the 2-D image obtained at step 1000 are identified.At 1006, key points on the 2-D image of the individual (user) obtainedat step 1002, are identified.

At 1008, the system modifies the key points on the 2-D image to be usedas a mask, based on the key points identified for the 2-D image of theindividual (user). The positioning of key points of the mask image aremodified to match the positioning of key points of the individual'simage. The modified image of the mask is then applied by the system onthe image of the face of the individual. In another embodiment, thefront-face image of the individual, comprised within the key points, arereplaced by the modified mask-image.

The system is therefore capable of rapidly generating a masked image ofa 2-D face. In this embodiment, AR masks are created for eachconsecutive frame, or each frame after a pre-defined number of frames,obtained from a video captured through a camera or taken from a videogallery. In an embodiment, the system uses a combination of programmaticinstructions to identify frames from a video and use them to processaccording to the steps described above in context of FIG. 10. In theembodiment, AR masks are created on a frame by frame basis, therebyallowing for the generation of a plurality of facial images, eachcorresponding to one of the frames. In various embodiments, the systemcan superimpose any other image, such as glasses, hats, crazy eyes,facial hair, on each image, thereby creating a video feed with AR.

FIG. 11 illustrates multiple AR images 1102, 1104, 1106, 1108, 1110,1112, 1114, 1116, and 1118, created with the method outlined withreference to FIG. 10. Heart-shaped figures overlay eyes of a user, asseen in image 1102. In an embodiment, the heart-shaped figures mayshrink each time the user blinks, and expand when the user has openeyes. As shown in image 1106, an AR image of a rainbow-like vomit fallsout of the user's mouth each time the user opens the mouth. Images 1114to 1118 illustrate face of the user that has been augmented by faces ofdifferent individuals. In embodiments, the faces of differentindividuals may be sourced from one or more digital databases.

FIG. 12 is a flow chart illustrating an exemplary computer-implementedmethod to convert a 2-D image into a modified image that has increaseddimensional depth (and, therefore, appears three dimensional to a user)for display of faceless gaming and other interactive display characters,which are controlled by real individuals. These may include users ofHeads-up Displays (HUDs) and players involved in multiplayer games. Inembodiments, 3-D images of the users/players are recreated withexpressions and/or reactions within an interactive virtual reality,augmented reality, or mixed reality environment, such as but not limitedto chatting and gaming environments. In embodiments, theexpressions/reactions of the users/individuals/players are tracked inreal time and thereby reflected through their corresponding 3-D imagesseen on a display. In this case, a 2-D frontal face image of theindividual is used to create a replica of at least the face, in 3-D. At1200, the system obtains an image of the individual from one of thesources including, but not limited to, an independent video camera, or avideo camera integrated with a mobile or any other computing device.

At 1202, the system, according to various embodiments of the presentspecification, identifies key points on the 2-D image. In an embodiment,the system uses a plurality of programmatic instructions designed tographically identify a plurality of key points, to identify at least 67key points. Subsequently the system derives a texture map for the 2-Dimage. The system derives a texture map using the following steps. At1204, the system identifies a plurality of non-overlapping, three-pointregions based on the identified key points. The system uses Delaunaytriangulation 1205 to define the three-point regions, based on theidentified key points, as described above. At 1206, the system projectsthe triangulated 2-D image on UV map of the generic 3-D mesh model. At1208, the system determines one or more positional relationships withinthe 2-D image. As described above, the positional relationships comprisea plurality of distances between anatomical features in the facialimage, and ratios of those distances to a specific anatomical distancesuch as temple width, which are necessarily unique to the 2-D image.Similarly, the system determines one or more positional relationshipswithin the generic 3-D mesh model of a face. As described above, thepositional relationships comprise a plurality of proportions that arestandard for a generic 3-D face model and comprise a plurality ofdistances between anatomical features in the 3-D face model, and ratiosof those distances to a specific anatomical distance such as templewidth, which define the generic 3-D face model.

At 1210, the system then uses proportions for the 2-D image and thecorresponding proportions from the generic 3-D image to determine thescaling factors. At 1212, the system adjusts the 3-D model based on thedetermined scaling factors and at 1214, the system modifies 3-D displayof the face based on expressions and/or reactions of the individual.

FIG. 13 illustrates exemplary images 1304 and 1306 captured from gamingdisplays. Image 1304 shows expressions/reactions of a player createdusing an embodiment of the present specification, and displayed as image1302 of an otherwise faceless First Person Shooter (FPS) character ingame using HUD. In an alternative multiplayer gaming environment 1306,expressions/reactions of four players in a game are each seen in images1308, 1310, 1312, and 1314.

FIG. 14 is a flow chart illustrating an exemplary computer-implementedmethod to convert a 2-D image in to a 3-D image for display of playersin gaming environments and other interactive display characters, whichare controlled by real individuals. In embodiments, 3-D images of theusers/players are recreated with expressions and/or reactions within aninteractive virtual reality, augmented reality, or mixed realityenvironment, such as but not limited to chatting and gamingenvironments. In embodiments, the expressions/reactions of theusers/individuals/players are tracked in real time and thereby reflectedthrough their corresponding 3-D images seen on a display. In this case,a 2-D frontal face image of the individual is used to create a replicaof at least the face, in 3-D. At 1400, the system obtains an image ofthe individual from one of the sources including, but not limited to, anindependent camera, a camera integrated with a mobile or any othercomputing device, or an image gallery accessible through a mobile or anyother computing device.

At 1402, the system, according to various embodiments of the presentspecification, identifies key points on the 2-D image. In an embodiment,the system uses a plurality of programmatic instructions designed tographically identify a plurality of key points, to identify at least 67key points. Subsequently the system derives a texture map for the 2-Dimage. The system derives a texture map using the following steps. At1404, the system identifies a plurality of non-overlapping, three-pointregions based on the identified key points. The system uses Delaunaytriangulation 1405 to define the three-point regions, based on theidentified key points, as described above. At 1406, the system projectsthe triangulated 2-D image on UV map of the generic 3-D mesh model. At1408, the system determines one or more positional relationships withinthe 2-D image. As described above, the positional relationships comprisea plurality of distances between anatomical features in the facialimage, and ratios of those distances to a specific anatomical distancesuch as temple width, which are necessarily unique to the 2-D image.Similarly, the system determines one or more positional relationshipswithin the generic 3-D mesh model of a face. As described above, thepositional relationships comprise a plurality of proportions that arestandard for a generic 3-D face model and comprise a plurality ofdistances between anatomical features in the 3-D face model, and ratiosof those distances to a specific anatomical distance such as templewidth, which define the generic 3-D face model.

At 1410, the system then uses proportions for the 2-D image and thecorresponding proportions from the generic 3-D image to determine thescaling factors. At 1412, the system adjusts the 3-D model based on thedetermined scaling factors and at 1414, the 3-D display of the face ismodified based on expressions and/or reactions and movements of theindividual.

In an embodiment, the system is capable of rapidly generating a 3D imageof a 2D face. In this embodiment, 3D images are created for eachconsecutive frame, or each frame after a pre-defined number of frames,obtained from a video captured through the camera. In an embodiment, thesystem uses a combination of programmatic instructions to identifyframes from the video and use them to process according to the stepsdescribed above in context of FIG. 14. In the embodiment, facialexpressions of a user are recreated through their 3D images on a frameby frame basis, thereby allowing for the generation of a plurality of 3Dfacial expressions, each corresponding to one of the frames.

FIG. 15 illustrates a still image 1502 from a display of a gamingapplication, where a 3-D image 1504 of a user (player) is seen in thedisplay. In embodiments, the expressions/reactions and movements ofimage 1504 reflect the user's expressions/reactions and movements inreal time.

FIG. 16 is a flow chart illustrating an exemplary computer-implementedmethod to convert a 2-D image of an individual in to a 3-D image thatmay be used to print one or more personalized avatars of the individual.In this case, a 2-D frontal face image of the individual is used tocreate a replica of at least the face, in 3-D. The recreated face in 3-Dmay then be combined with an image of a body to create a personalizedavatar which can be printed using 3-D printing methods. At 1600, thesystem obtains an image of the individual from one of the sourcesincluding, but not limited to, an independent camera, a cameraintegrated with a mobile or any other computing device, or an imagegallery accessible through a mobile or any other computing device.

At 1602, the system, according to various embodiments of the presentspecification, identifies key points on the 2-D image. In an embodiment,the system uses a plurality of programmatic instructions designed tographically identify a plurality of key points, to identify at least 67key points. Subsequently the system derives a texture map for the 2-Dimage. The system derives a texture map using the following steps. At1604, the system identifies a plurality of non-overlapping, three-pointregions based on the identified key points. The system uses Delaunaytriangulation 1605 to define the three-point regions, based on theidentified key points, as described above. At 1606, the system projectsthe triangulated 2-D image on UV map of the generic 3-D mesh model. At1608, the system determines one or more positional relationships withinthe 2-D image. As described above, the positional relationships comprisea plurality of distances between anatomical features in the facialimage, and ratios of those distances to a specific anatomical distancesuch as temple width, which are necessarily unique to the 2-D image.Similarly, the system determines one or more positional relationshipswithin the generic 3-D mesh model of a face. As described above, thepositional relationships comprise a plurality of proportions that arestandard for a generic 3-D face model and comprise a plurality ofdistances between anatomical features in the 3-D face model, and ratiosof those distances to a specific anatomical distance such as templewidth, which define the generic 3-D face model.

At 1610, the system then uses proportions for the 2-D image and thecorresponding proportions from the generic 3-D image to determine thescaling factors. At 1612, the system adjusts the 3-D model based on thedetermined scaling factors and at 1614, the system prints a personalizedavatar of the 3-D display of the face. In embodiments, printing isperformed using 3-D printing methods. FIG. 17 illustrates some examplesof different (four) avatars of different users printed using embodimentsof the process described in context of FIG. 16.

FIG. 18 is a flow chart illustrating an exemplary computer-implementedmethod to convert a 2-D image in to a 3-D image for driving key frameanimation based on facial expressions of an individual. In embodiments,the expressions/reactions of the users/individuals/players are trackedin real time and thereby reflected through their corresponding 3-Dimages seen on a display, which in turn is used to drive key frameanimation. In this case, a 2-D frontal face image of the individual isused to create a replica of at least the facial expressions andmovements of an animated figure, in 3-D. In embodiments, the animatedfigure could be one of the 3D image of the face of the user, an object,or any other face created for the animation. At 1800, the system obtainsan image of the individual from one of the sources including, but notlimited to, an independent video camera, a video camera integrated witha mobile or any other computing device, or a video gallery accessiblethrough a mobile or any other computing device.

At 1802, the system, according to various embodiments of the presentspecification, identifies key points on the 2-D image. In an embodiment,the system uses a plurality of programmatic instructions designed tographically identify a plurality of key points, to identify at least 67key points. Subsequently the system derives a texture map for the 2-Dimage. The system derives a texture map using the following steps. At1804, the system identifies a plurality of non-overlapping, three-pointregions based on the identified key points. The system uses Delaunaytriangulation 1805 to define the three-point regions, based on theidentified key points, as described above. At 1806, the system projectsthe triangulated 2-D image on UV map of the generic 3-D mesh model. At1808, the system determines one or more positional relationships withinthe 2-D image. As described above, the positional relationships comprisea plurality of distances between anatomical features in the facialimage, and ratios of those distances to a specific anatomical distancesuch as temple width, which are necessarily unique to the 2-D image.Similarly, the system determines one or more positional relationshipswithin the generic 3-D mesh model of a face. As described above, thepositional relationships comprise a plurality of proportions that arestandard for a generic 3-D face model and comprise a plurality ofdistances between anatomical features in the 3-D face model, and ratiosof those distances to a specific anatomical distance such as templewidth, which define the generic 3-D face model.

At 1810, the system then uses proportions for the 2-D image and thecorresponding proportions from the generic 3-D image to determine thescaling factors. At 1812, the system adjusts the 3-D model based on thedetermined scaling factors and at 1814, the system drives key frameanimation using the adjusted 3-D display of the face.

In an embodiment, the system is capable of rapidly generating a 3D imageof a 2D face. In this embodiment, key frame animations are created foreach consecutive frame, or each frame after a pre-defined number offrames, obtained from the video captured through the camera or takenfrom the video gallery. In an embodiment, the system uses a combinationof programmatic instructions to identify frames from the video and usethem to process according to the steps described above in context ofFIG. 18. In the embodiment, key frame animations are created on a frameby frame basis, thereby allowing for the generation of a plurality of 3Dfacial images including expressions and movements of the user, eachcorresponding to one of the frames. In embodiments, additional animationeffects are superimposed on the frames.

The above examples are merely illustrative of the many applications ofthe system of present invention. Although only a few embodiments of thepresent invention have been described herein, it should be understoodthat the present invention might be embodied in many other specificforms without departing from the spirit or scope of the invention.Therefore, the present examples and embodiments are to be considered asillustrative and not restrictive, and the invention may be modifiedwithin the scope of the appended claims.

We claim:
 1. A computer-implemented method for increasing a dimensionaldepth of a two-dimensional image of a face to yield a face image thatappears three dimensional, said method being implemented in a computerhaving a processor and a random access memory, wherein said processor isin data communication with a display and with a storage unit, the methodcomprising: acquiring from the storage unit the two-dimensional image ofthe face; acquiring from the storage unit a three-dimensional meshimage; using said computer and executing a plurality of programmaticinstructions stored in the storage unit, identifying a plurality of keypoints on the two-dimensional image of the face; using said computer andexecuting a plurality of programmatic instructions stored in the storageunit, generating a texture map of the two-dimensional image of the face;using said computer and executing a plurality of programmaticinstructions stored in the storage unit, projecting said texture map ofthe two-dimensional image of the face onto the three-dimensional meshimage; using said computer and executing a plurality of programmaticinstructions stored in the storage unit, determining a first set of oneor more proportions within the two-dimensional image of the face; usingsaid computer and executing a plurality of programmatic instructionsstored in the storage unit, determining a second set of one or moreproportions within the three-dimensional mesh image; using said computerand executing a plurality of programmatic instructions stored in thestorage unit, determining a plurality of scaling factors, wherein eachof said scaling factors is a function of one of said first set of one ormore proportions and a corresponding one of said second set of one ormore proportions; using said computer and executing a plurality ofprogrammatic instructions stored in the storage unit, adjusting thethree-dimensional mesh image based on the determined plurality ofscaling factors to yield the face image that appears three dimensional;and using said computer, outputting the face image that appears threedimensional.
 2. The computer-implemented method of claim 1, wherein thekey points include points representative of a plurality of anatomicallocations on the face, wherein said anatomical locations include pointslocated on the eyebrows, eyes, nose, and lips.
 3. Thecomputer-implemented method of claim 1, wherein the texture mapcomprises a plurality of non-overlapping, triangular regions.
 4. Thecomputer-implemented method of claim 1, wherein each of said pluralityof scaling factors is a ratio of one of said first set of one or moreproportions to the corresponding one of said second set of one or moreproportions.
 5. The computer-implemented method of claim 1, wherein thedetermining the first set of one or more proportions within thetwo-dimensional image comprises determining proportions frommeasurements between at least two anatomical positions on the face. 6.The computer-implemented method of claim 1, wherein the determining afirst set of one or more proportions within the two-dimensional imagecomprises determining a first anatomical distance and dividing saidfirst anatomical distance by a second anatomical distance.
 7. Thecomputer-implemented method of claim 6, wherein the first anatomicaldistance is at least one of a lateral face width, a lateral jaw width, alateral temple width, a lateral eyebrow width, a lateral chin width, alateral lip width, and a lateral nose width and wherein the secondanatomical distance is a distance between two temples of the face. 8.The computer-implemented method of claim 6, wherein the first anatomicaldistance is at least one of a vertically defined lip thickness, avertical distance between a nose and a nose bridge, a vertical distancebetween a lip and a nose bridge, a vertical distance between a chin anda nose bridge, a vertical eye length, and a vertical distance between ajaw and a nose bridge and wherein the second anatomical distance is atleast one of a distance between two anatomical positions on said faceand a distance between two temples of the face.
 9. Thecomputer-implemented method of claim 6, wherein the first anatomicaldistance is a distance between two anatomical positions on said face andthe second anatomical distance is a distance between a point locatedproximate a left edge of a left eyebrow of the face and a point locatedproximate a right edge of a right eyebrow of the face.
 10. Thecomputer-implemented method of claim 1, wherein the determining a secondset of one or more proportions within the three-dimensional mesh imagecomprises determining a first anatomical distance and dividing saidfirst anatomical distance by a second anatomical distance.
 11. Thecomputer-implemented method of claim 10, wherein the first anatomicaldistance is at least one of a lip thickness, a distance between a noseand a nose bridge, a distance between a lip and a nose bridge, adistance between a chin and a nose bridge, an eye length, and a distancebetween a jaw and a nose bridge of the three-dimensional mesh image andwherein the second anatomical distance is a distance between twoanatomical positions on said three-dimensional mesh image.
 12. Thecomputer-implemented method of claim 10, wherein the first anatomicaldistance is a distance between two anatomical positions on saidthree-dimensional mesh image and the second anatomical distance is adistance between a point located proximate a left edge of a left eyebrowof the three-dimensional mesh image and a point located proximate aright edge of a right eyebrow of the three-dimensional mesh image. 13.The computer-implemented method of claim 1, wherein, prior toidentifying the plurality of key points on the two-dimensional image ofthe face, processing the two-dimensional image to validate a presence ofa frontal image of the face.
 14. A computer readable non-transitorymedium comprising a plurality of executable programmatic instructionswherein, when said plurality of executable programmatic instructions areexecuted by a processor, a process for increasing a dimensional depth ofa two-dimensional image of a face to yield a face image that appearsthree dimensional is performed, said plurality of executableprogrammatic instructions comprising: programmatic instructions, storedin said computer readable non-transitory medium, for acquiring from thestorage unit the two-dimensional image of the face; programmaticinstructions, stored in said computer readable non-transitory medium,for acquiring from the storage unit a three-dimensional mesh image;programmatic instructions, stored in said computer readablenon-transitory medium, for identifying a plurality of key points on thetwo-dimensional image of the face; programmatic instructions, stored insaid computer readable non-transitory medium, for generating a texturemap of the two-dimensional image of the face; programmatic instructions,stored in said computer readable non-transitory medium, for translatingsaid texture map of the two-dimensional image of the face onto thethree-dimensional mesh image; programmatic instructions, stored in saidcomputer readable non-transitory medium, for determining a first set ofone or more proportions within the two-dimensional image of the face;programmatic instructions, stored in said computer readablenon-transitory medium, for determining a second set of one or moreproportions within the three-dimensional mesh image; programmaticinstructions, stored in said computer readable non-transitory medium,for determining a plurality of scaling factors, wherein each of saidscaling factors is a function of one of said first set of one or moreproportions and a corresponding one of said second set of one or moreproportions; and programmatic instructions, stored in said computerreadable non-transitory medium, for adjusting the three-dimensional meshimage based on the determined plurality of scaling factors to yield theface image that appears three dimensional.
 15. The computer readablenon-transitory medium of claim 14, wherein the key points include pointsrepresentative of a plurality of anatomical locations on the face,wherein said anatomical locations include points located on theeyebrows, eyes, nose, and lips.
 16. The computer readable non-transitorymedium of claim 14, wherein the texture map comprises a plurality ofnon-overlapping, triangular regions.
 17. The computer readablenon-transitory medium of claim 14, wherein the determining one or moreproportions within the two-dimensional image comprises determiningproportions from measurements between at least two anatomical positionson the face.
 18. The computer readable non-transitory medium of claim14, wherein each of said plurality of scaling factors is a ratio of oneof said first set of one or more proportions to the corresponding one ofsaid second set of one or more proportions.
 19. The computer readablenon-transitory medium of claim 14, wherein the determining a first setof one or more proportions within the two-dimensional image comprisesdetermining a first anatomical distance and dividing said firstanatomical distance by a second anatomical distance.
 20. The computerreadable non-transitory medium of claim 19, wherein the first anatomicaldistance is at least one of a lateral face width, a lateral jaw width, alateral temple width, a lateral eyebrow width, a lateral chin width, alateral lip width, and a lateral nose width and wherein the secondanatomical distance is a distance between two temples of the face. 21.The computer readable non-transitory medium of claim 19, wherein thefirst anatomical distance is at least one of a vertically defined lipthickness, a vertical distance between a nose and a nose bridge, avertical distance between a lip and a nose bridge, a vertical distancebetween a chin and a nose bridge, a vertical eye length, and a verticaldistance between a jaw and a nose bridge and wherein the secondanatomical distance is a distance between two temples of the face. 22.The computer readable non-transitory medium of claim 19, wherein thefirst anatomical distance is a distance between two anatomical positionson said face and the second anatomical distance is a distance between apoint located proximate a left edge of a left eyebrow of the face and apoint located proximate a right edge of a right eyebrow of the face. 23.The computer readable non-transitory medium of claim 14, wherein thedetermining a second set of one or more proportions within thethree-dimensional mesh image comprises determining a first anatomicaldistance and dividing said first anatomical distance by a secondanatomical distance.
 24. The computer readable non-transitory medium ofclaim 23, wherein the first anatomical distance is at least one of a lipthickness, a distance between a nose and a nose bridge, a distancebetween a lip and a nose bridge, a distance between a chin and a nosebridge, an eye length and a distance between a jaw and a nose bridge ofthe three-dimensional mesh image and wherein the second anatomicaldistance is a distance between two anatomical positions on saidthree-dimensional mesh image.
 25. The computer-implemented method ofclaim 10, wherein the first anatomical distance is a distance betweentwo anatomical positions on said three-dimensional mesh image and thesecond anatomical distance is a distance between a point locatedproximate a left edge of a left eyebrow of the three-dimensional meshimage and a point located proximate a right edge of a right eyebrow ofthe three-dimensional mesh image.
 26. The computer readablenon-transitory medium of claim 14, further comprising programmaticinstructions, stored in said computer readable non-transitory medium,for processing the two-dimensional image to validate a presence of afrontal image of the face prior to identifying the plurality of keypoints on the two-dimensional image of the face.